feat: Add task_kwargs param to dbt_flow #47
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kiyanmair
commented
Jul 14, 2024
| @@ -11,6 +11,7 @@ def dbt_flow( | |||
| profile: Optional[DbtProfile] = None, | |||
| dag_options: Optional[DbtDagOptions] = None, | |||
| flow_kwargs: Optional[dict] = None, | |||
| task_kwargs: Optional[dict] = None, | |||
There was a problem hiding this comment.
I noticed that in tasks.py, Dict is imported from typing (along with List), while flow.py just uses dict. If this is an inconsistency, I'm happy to clean it up in this PR.
| @@ -237,6 +239,7 @@ def generate_tasks_dag( | |||
| profile=profile, | |||
| dag_options=dag_options, | |||
| dbt_node=dbt_node, | |||
| task_kwargs=task_kwargs, | |||
There was a problem hiding this comment.
For now I've applied the same task_kwargs to all dbt tasks corresponding to resource types, but not those corresponding to dbt tests (below). However, I'm wondering if a more specific method of supplying task_kwargs is preferred, for example to specify different values for tests or even for different resource types.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The functions which create Prefect tasks for all dbt resource types (
_task_dbt_snapshot,_task_dbt_seed, and_task_dbt_run) all have optionaltask_kwargsparameters, but these don't seem to be used. It would be useful to be able to set these kwargs when creating a dbt flow, for example to specify retry behaviour.This PR simply adds the
task_kwargsparameter todbt_flowand passes its value through togenerate_tasks_dag. Within the latter, this PR suppliestask_kwargswhen creatingall_tasks, which applies to the tasks corresponding to resource types only.Resolves #42.
NOTE: I haven't tested this change yet. I'm also new to this repo and may have overlooked something.